AUSTRALIAN UNIVERSITIES' REVIEW 

Please don't aim for a 
highly cited paper 

Michael C Calver 

Murdoch University 

Citation-based metrics are important in determining careers, so it is unsurprising that recent publications advise prospective authors 
on how to write highly cited papers. While such publications offer excellent advice on structuring and presenting manuscripts, there are 
significant downsides, including: restrictions in the topics researched, incentives to misconduct and possible detriments to motivation, 
innovation and collegiality. Guides to writing highly cited papers also assume that all citations are equal, ignoring new directions in 
bibliometric research identifying ‘quality’ and perfunctory citations. Rather than pursuing citations, with the uncertainty about their 
significance and the potential negative consequences, authors may fare better by following evidence from several disciplines indicating that 
persistence, a focused research program, good methodology and publishing in relevant journals are more important in career development 
and disciplinary influence than the odd star paper. Research administrators could encourage such steps by considering innovative new 
multivariate assessments of research productivity, including assessing social impact. 
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Introduction 

Increasingly, researchers find their track records under 
scrutiny as supervisors and funding agencies seek the 
most successful groups to fund, or individuals to reward 
with appointments, promotion, tenure or prizes (Corsi, 
D’Ippoliti & Lucidi, 2010; Oswald, 2010; Teixeira et at., 
2013). Consulting the citation records in major databases 
is a more rapid and cheaper way to make such assessments 
than through peer review (Hodge & Lacasse, 2011; Buela- 
Casal & Zych, 2012), so citation profiles are increasingly 
important in determining careers (Calver, 2013; Calver et 
at, 2013a; vanWesel.Wyaat & ten Haaf, 2014). 

One must publish to be cited, so it is unsurprising that 
the valuable guides to scientific writing and publishing 
from experienced authors, editors or teachers (e.g. 
Glasman-Deal, 2010; Cargill & O’Connor, 2013;Tress,Tress 
& Saunders, 2014) have been joined by a new category 
of advice - how to write a highly cited paper (Pyke 
2013; 2014). While both traditional and new guides share 
concerns for excellent presentation, sound scholarship 
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and clarity of style, encouraging people to aim 
consciously to write highly cited papers is concerning 
for five main reasons: (i) it narrows the scope of research 
undertaken or published; (ii) the focus on reward 
may reduce intrinsic motivation, innovation and true 
collaboration but encourage mistakes and misconduct; 
(iii) despite substantial research, the significance of 
citations remains controversial; (iv) empirical evidence 
from several disciplines indicates that persistence, 
research specialisation, good methodology and 
publishing in germane journals are most important in 
developing a career and influencing a discipline, not the 
occasional highly cited paper; and (v) the convenience of 
citation counting may impede development or adoption 
of innovative new multivariate assessments of research 
productivity, including evaluation of wider social impact. 

Narrowing research scope 

The topic and the type of study are important predictors 
of citation potential, so choosing a topic is a key part of 
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advice on writing highly cited papers (Pyke, 2013; 2014). 
Taxonomists, for example, have vented their frustration 
regarding belittling of their important work on the basis 
of the low citations it attracts (Valdecasas, Castroviejo 
& Marcus, 2000; Valdecasas, 2014). Their specialised 
subject matter is of immediate interest to just a few 
experts (Ebach, Valdecasas & Wheeler, 2011), although 
it subsequently underpins (without citation) diverse 
areas of biological inquiry (MacRoberts & MacRoberts, 
2010). The same is true of many areas in mathematics 
(Adler, Ewing & Taylor, 2008), some medical specialities 
(van Eck et al , 2013) and numerous applied disciplines 
where the significance of work comes not from academic 
citations but from influence on practitioners (Shewan & 
Coats, 2006;Jones, 2007; Calver, Lilith & Dickman, 2013b). 

In contrast, hot topics and reviews are more likely to 
attract citations (Davis et al., 2008; Davis, 2009;Teixeira 
et al., 2013), as are positive studies (those supporting the 
hypothesis tested) (Falagas &Alexiou, 2008; Song et al, 
2010;Jannot et al, 2013). Keeping good company may be 
important too: papers with many authors or at least one 
highly cited author attract more citations, which is why it 
is necessary to control for the number of authors in some 
bibliometric studies (Calver & Bradley, 2010). 

Thus authors hungry for citations will neglect some 
fields or topics, regardless of their potential merit, in 
favour of the fashionable or the review, preferably written 
with many influential colleagues. They certainly won’t 
tackle ... areas of public interest where little research is 
being done' (Martin, 2012, p. 168) and will probably have 
less time for applied research or engaging the public 
(B. Martin, 2011). A narrowing of research diversity is as 
valuable to scholarship as atherosclerosis is to the cardiac 
patient, yet one follows citation hunger as surely as the 
other follows a fatty diet. 

Focus on reward 

Intrinsic motivation, or the desire to complete a task 
for personal satisfaction rather than external reward 
(Cerasoli, Nicklin & Ford, 2014), may be at risk when 
the focus is on an external outcome or reward, such as 
high citations, rather than the process of conceiving, 
implementing and communicating a good study. Some 
empirical studies and meta-analyses of relationships 
between external rewards and intrinsic motivation find 
a reduction in intrinsic motivation when rewards are 
offered (e.g. Ryan & Deci, 1996; Deci & Ryan, 2013; Green, 
2014). Others do not (e g. Eisenberger & Cameron, 1996; 
Reiss, 2005). Nevertheless, even critics of negative links 
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between external rewards and intrinsic motivation accept 
that rewards depress intrinsic motivation when the task is 
interesting, the possibility of reward is known in advance 
and the likelihood of reward is ambiguously connected 
to performance (Cameron, Banko & Pierce, 2001). This 
matches aiming for a highly cited paper because the 
research is interesting, authors know that citations may 
follow, but even good work may only receive modest 
citations (Valdecasas et al, 2000; Shewan & Coats, 2006; 
Jones, 2007). 

Critiques of using rewards as motivation also note that 
while they may increase basic productivity, innovation 
may decline (but see Curran & Walsworth, 2014 for 
suggestions that the type of reward may be important in 
determining whether or not it encourages innovation). 
It may be better to use tried and true methods in case 
the new approach fails, or people may simply be so 
busy chasing the reward that they do not consider an 
alternative approach (Ariely et al , 2009(Webb .Williamson 
& Zhang, 2013). Charlton (2008) speculated that science 
in the UK is undergoing just such a process, with a 
preference for more productive normal science’ over 
less productive but potentially more ground-breaking 
revolutionary science.’ In academia, the problem may be 
compounded by a preference for the tried and true at 
the grant application stage (Martin, 2000) or in editorial 
process with publications (Horrobin, 1990). Surprisingly, 
mistakes may actually increase when the focus is on the 
outcome rather than the process, a phenomenon well 
documented in declines in athletic performance under 
pressure (DeCaro et al , 2011). 

True collaboration may suffer too, because genuine 
collaborative relationships may be replaced by a 
‘contrived collegiality’ in which there is uncritical 
acceptance of striving for an external goal or meeting an 
external agenda (Boocock, 2011).For example,in 2007 the 
UK began planning its first iteration of the REF (Research 
Evaluation Framework) for evaluating research in higher 
education, with a strong emphasis on citations. Several 
senior academics predicted that authorship practices 
would shift to the disadvantage of junior researchers. 
Specifically, junior researchers would be excluded from 
secondary authorship on group publications so that 
they could later cite those papers without a penalty for 
self-citation for the senior colleagues on the original 
papers. On the other hand, senior researchers were 
predicted to form citation clubs’, in which they agreed 
to cite each other’s papers for mutual benefit (Corbyn, 
2008). Citation clubs are already documented at the 
journal, if not the individual, level (van Noorden, 2013). 
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These examples fulfil predictions that obsession with 
numerical assessments rewards aggressive, acquisitive 
and exploitative behaviour (Lawrence, 2002). 

Finally, a conviction that success is deserved’ may 
tempt authors into misconduct to gain the reward, just 
as an athlete may rationalise doping (Martinson,Anderson 
& De Vries, 2005; Fanelli, 2009; Deci & Ryan, 2013). 
Thus Excessive focus on building publication records 
may lead to over-bias towards choosing high-impact 
journals for publication, violation of publication ethics, 
and unbalanced development of writing skills’ (Cheung, 
2008, p. 41). Buela-Casal (2014) argues that excessive 
concern with research status and associated marketing 
meet the criteria of a psychological disorder, manifesting 
in behaviour such as fraudulent misrepresentation of 
research data or bibliometric profiles, personal belief in 
such misrepresentations, and an obsession with self-image. 

What does a citation mean? 

Critiques of simplistic citation analyses highlight 
numerous biases and unchecked assumptions, some of 
which remain unaddressed (Appendix 1). Furthermore, 
in some fields citations correlate significantly with peer 
assessments of impact or influence such as prizes and 
awards, but in others they do not (Bergsma, Mandryk & 
McCalla, 2014). This calls into question their validity as 
measures of quality. 

More recently, bibliometricians note that not all citations 
are equal, even leading to the farcical situation where a 
non-existent paper is cited frequently because authors 
have copied a referencing error (Dubin, 2004). Not so 
farcical are examples where misrepresentation of what 
was actually written or echoing a fallacy by not checking 
the original promulgates misinformation (Wetterer, 
2006; Wright & Armstrong, 2008). These problems may 
be common. In the field of marine biology one in four 
citations was found to be ambiguous in relation to the 
statement it was supposed to support, offered no support 
at all, or was empty (a reference to a secondary source) 
(Todd et al., 2010). 

A recent review claims that between 40 and 80 per cent 
of citations are inessential or perfunctory. The authors 
then suggest a framework for identifying influential 
citations on the basis of repeated use in one paper, 
similarities between the citation title and the title or sub¬ 
sections of the citing paper, the context of the citation 
in the paper (noting any descriptors such as ‘important’, 
baseline’, key’ etc. or whether the citation appears alone 
or in a series), and the location of the citation in the paper 
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(Zhu et al ., 2014). Bergsma et al. (2014, p. 35) also claim 
significant improvement in assessing an author’s influence 
by replacing citations with multivariate data including 
novel social, linguistic, psychological, and bibliometric 
features .’Applying methods such as these may reduce the 
value of some highly cited works such as reviews relative 
to empirical papers making original contributions. They 
would also threaten questionable practices such as the 
boosting of citations by commissioning opinion pieces 
that offer extensive opportunities for the self-citation of 
authors and journals (Heneberg, 2014). 

Overall, striving for a highly cited paper is not necessarily 
the same as aiming to produce influential work. To use 
an analogy from novelists, Harold Robbins may have sold 
more copies than William Faulkner or Ernest Hemingway, 
but Faulkner and Hemingway have surely had a greater 
influence on modern literature. 

Lessons from successful researchers 

Commentaries by authors on their highly cited papers 
nominated interest from the scientific and lay communities, 
novelty, utility to broad audiences and significance in 
addressing an important or fundamental problem as key 
reasons for the high citations. The categories were not 
mutually exclusive, so authors often referred to more than 
one (Small, 2004). 

However, significant influence on a discipline is more 
likely to follow from consistent performance over a 
career, not one star paper. In ecology, such influence 
correlates strongly with research specialisation, a 
focused body of work (not necessarily on a fashionable 
topic) and publication of good papers in germane (not 
necessarily highly ranked) journals (Parker, Allesina 
& Lortie, 2013). In this context, it is worth noting that 
the citation gap between elite journals and the others 
is declining (Acharya et al., 2014). Padial et al. (2010) 
further stressed the value of innovation in ecological 
papers. Hermanowicz (2006), in a survey of physicists, 
found that the top-ranked quality for success was 
persistence in the face of difficulty or publication 
rejection. This is echoed in the case of ecology by 
Cassey and Blackburn (2003, p. 375), who found that 
publication success and manuscript rejection are not 
strangers’, and in psychology by Douglas (1992), who 
claimed to have received up to three rejections for every 
highly cited paper he published. To all of these can be 
added the blessings of Lady Luck, because of ‘... the key 
role that error, chance or accident can sometimes play in 
scientific endeavour’ (Campanario, 1996, p. 20). 

Please don't aim for a highly cited paper Michael C Calver 47 


AUSTRALIAN 


UNIVERSITIES 


REVIEW 


Innovations in research assessment 

Citation counting for research assessment is attractive to 
research administrators because it is speedy and offers the 
illusion of objective numbers (Adler et al., 2008), despite 
the problems reported (Appendix 1). This may restrain 
support for the more challenging approach of allowing 
individuals or organisations undergoing evaluation to 
demonstrate the impact of their work beyond bibliometric 
indicators. Possibilities might be changes in professional 
or government practice as a result of research (Witten & 
Hammond, 2010), publishing in local languages rather than 
English to reach regional communities or practitioners 
(Adler & Harzing, 2009), or online readership/download 
statistics where readership may be more important than 
citations (Bollen et al., 2009). 

Several major initiatives are attempting just such broad 
assessments. For example, STAR METRICS (Science and 
Technology for America’s Reinvestment: measuring the 
effects of research on innovation, competitiveness, and 
science) focuses on a wide range of results and impacts 
from federally-funded US researchers (Largent & Lane, 
2012). In Brazil, the Lattes Platform takes a multivariate 
statistical approach to evaluate the broad impact of 
Brazilian researchers (Araujo et al, 2014). In Europe, the 
Social Impact Assessment Methods through Productive 
Interactions (SIAMPI) project is a broad-based initiative 
involving contributors fromThe Netherlands,Spain,France 
and the UK. Its goals include assessing the social impact 
of research and providing granting bodies with diverse 
data on the effectiveness of their activities (Molas-Gallart 
& Tang, 2011; Spaapen & van Drooge, 2011). Innovative 
online resources are also growing. Examples include: 
Metrics from Scholarly Usage of Resources (MESUR), 
Standardised Usage Statistics Harvesting Initiative 
(SUSHI), Shared E-Resource Understanding (SERU) 
and Counting Online Usage of NeTworked Electronic 
Resources (COUNTER) (Pesch, 2011). These suggestions 
are difficult and possibly expensive (B R Martin, 2011), 
but the alternative is a simplistic, restricted assessment of 
the value of research to the community (Lane, 2010; Lane 
& Bertuzzi, 2011). 

Assessment 

While some assert that Writing highly cited articles is 
an important goal for scholars’ (van Wesel et al., 2014, 
p. 1602), aiming for a highly cited paper concentrates 
on an outcome, not the process of doing good research 
and disseminating the results. As Douglas (1992, p. 405) 
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noted cynically: In any event, if you were to set out in 
cold blood to write a highly cited article, your best bet 
would be to devise or revise a paper-and-pencil test of 
personality or motivation, improve on a commonly used 
method, coin a snappy new word or phrase, or think of 
a new way to apply statistics.’ With no intent at cynicism, 
van Wesel et al. (2014, p. 1612) advise: If scholars or their 
institutions want to contribute to scientific literature, and 
to be seen to contribute, and if they wish promote (sic) 
their individual and collective reputations in rankings and 
evaluations, they need to be aware of how the invisible 
hand in science works, and how it can be influenced. 
Form and style also influence how well individual scholars 
and their institutions fare in the global competition that 
scientific publication has become.' 

Such an outcome-driven approach reduces research 
diversity,encourages selfish behaviour or even misconduct, 
conflicts with empirical evidence about what practices 
are most likely to lead to a strong and lasting influence 
and might hamper the uptake of innovative new methods 
of assessing the significance of research, especially 
efforts to assess social impact. It leads to complaints that 
... dodgy evaluation criteria such as impact factors and 
citations are dominating minds, distorting behaviour and 
determining careers’ (Lawrence, 2007, p. R583) and that 
research rankings based partly on citations are ... causing 
biomedical scientists to focus more on their careers and 
less on understanding nature and disease’ (Lawrence 2002, 
p. 835). Valuable points about style and crafting a paper 
can be taken from advice on writing highly cited work 
(Pyke, 2013; 2014; van Wesel et al. 2014), but there are 
dangers in moving beyond that to selection of research 
topics on the basis of citation potential. Surely, one should 
advise authors to aspire to influencing their discipline 
through quality work rather than writing a highly cited 
paper - with all the uncertainty about what a highly cited 
paper actually means. 

Research administrators might also consider the 
advantages of more broad-based methods of research 
assessment, heeding the call of The San Francisco 
Declaration on Research Assessment (DORA) (http:// 
am.ascb.org/dora/) to ... consider the value and impact 
of all research outputs (including datasets and software) 
in addition to research publications, and consider a broad 
range of impact measures including qualitative indicators 
of research impact, such as influence on policy and 
practice.’ Such a focus on the broad impacts of research 
would assist authors in shifting their emphasis from the 
‘hot topics’ that feed citation hunger to a wider range of 
problems of social, political and environmental relevance. 
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Appendix 1. Criticisms of citation analysis taken from MacRoberts and MacRoberts (1996), 
together with responses in bibliometric research. 


Criticism 

Response 

Authors do not cite all their influences 

The criticism that authors cite only a fraction of their influences in their papers remains 
unanswered, while the range of reasons other than influence for choosing a citation is growing. 
These include, amongst others, availability (Harnad etal,, 2004), personal contacts (White, 

2001), and a preference for secondary sources (MacRoberts and MacRoberts, 1996). 

Citations are biased 

It is acknowledged that, for a range of reasons, authors cite selectively, introducing bias (Song 
et al., 2010). The possibility of replacing citations selected by authors with others based on 
electronic assessment of publication similarity (‘virtual scientometry') is possible but challenging 
(Amancio etal., 2012). 

Secondary sources are preferred 

Reviews attract disproportionately high citations (Davis et al., 2008; Davis, 2009, Teixeira et al., 
2013), so the reviewer takes credit really due to the authors of the primary papers. The trend for 
reviews to receive disproportionately more citations is so well established that it is controlled in 
citation-based studies (e.g. Calver and Bradley 2010). 

Informal influences are uncited 

Informal influences may be noted in acknowledgements or ‘personal communications’ but not a 
full citation, which means that they earn no credit. 

Citer motivation is unknown 

It is recognised that citation is not simply an acknowledgement of credit - many factors influence 
the choice of citations in a paper. Journal restrictions on the number of citations allowed in a 
paper is one good example! Another is the pressure editors may place on authors to cite papers 
from particular journals (Yu etal., 2014). 

Citation rates vary between disciplines 

There has been much work on addressing this, particularly in relation to journals, through 
indices such as SJR 1 and SNIP 2 . Nevertheless, fine-scale differences in citations can exist between 
sub-fields (e.g. the higher citations for basic medical research as opposed to clinical medical 
research - Opthof, 2011; van Eck et al. ,2013, or variations in citation rates across subdisciphnes 
in mathematics - Smolinsky and Lercher, 2012). 

Self-citation 

This is recognised as excessive (Aksnes, 2003), with options to exclude it provided in some 
databases. 

Audiences vary between disciphnes, 
but are generally unknown 

There has been much work on addressing this, particularly in relation to journals, through indices 
such as SJR 1 and SNIP 2 . 

Some sources are traditionally not 
cited 

It is acknowledged that some types of information are traditionally uncited (for example, floras) 
(MacRoberts and MacRoberts, 2010). 

Ignorance of the hterature 

It is acknowledged that the hterature in many fields is so vast that an exhaustive review cannot be 
undertaken, so relevant papers may be missed from ignorance. 

Data are biased 

Bibliometricians acknowledge that databases are selective, so citation performance may vary 
depending on the one used (Jacso, 2008). 

Databases have technical problems 

Despite the best efforts, there are errors in databases (Leydesdorff, 2007). Bibliometricians offer 
advice on tracking down orphan references (references with no master records in a database) 
and stray references (that cannot be linked to the master record in the database because of errors 
by the citing authors), but such errors may exert an unknown influence on citation analyses 
(Jacso, 2008). 


1 SCImago Journal Rank (see Colledge etal. 2010 for further explanation) 

2 Source Normalised Impact per Paper (see Colledge et al. 2010 for further explanation) 
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